Enabling Onboard Cloud Cover Assessment From Hyperspectral Data 
with Adaptive Computing Resources 


Marco A. Figueiredo 
Patrick H. Stakem 

June 1998 


Introduction 

This white paper establishes the feasibility of using off-the-shelf computing hardware 
with a known path to full space qualification to address a large set of user processing 
needs in the area of sensed data sets for Earth Observation data. It is no longer feasible to 
simply collect large volumes of data that are downlinked over already strained 
communication channels to large central archive and processing centers based on older 
generation computational resources. Hyperspectral instalments have only made these 
problems worse. Not only do they introduce many more bands of sensed data, but these 
data must be correlated to make sense of observed phenomenology. The choice is 
between not collecting the data versus generating useful data products close to the data 
source, and directly downlinking these to users in the field, in addition to and in parallel 
with the classical data collection, downlink, and Level 0/1/2/3 processing. In many cases, 
the end user dictates the receipt of immediate processed data products at a remote site. 
This is not cost effective or even feasible for most current or planned data processing 
facilities for downlinked sensed data. The approach has been to attempt to migrate level 
0/1 processing onboard the spacecraft. This approach has met with limited success due to 
the limited computational resources available, and the high cost of storage. One approach 
applies data reduction by selection of relevant data for processing and downloading, via 
adaptive computing techniques. 

This paper also addresses the area of extracting information from raw data onboard the 
spacecraft at the sensor, and establishes the feasibility of generating data products 
onboard for direct downlink in a timely manner by using proven hardware, architecture, 
methodology, and algorithms. This approach does not preclude or interfere with the 
normal collection and archiving of sensed data, but rather works with data tapped off the 
main stream, and processed in a parallel stream. The advantage of this approach is not 
only the timeliness of the downlinked data product, but in the ability to make onboard 
decisions about data value. This frees the processing pipeline to operate on only data of 
interest. For instance, it may not always be appropriate to take data, particularly when 
cloud cover obscures the areas of interest in the visible region. The decision to take the 
data and downlink it can be made onboard, with an appropriate cloud assessment 
algorithm. We are assuming that neither the onboard data storage resources, the downlink 
bandwidth, nor the front end processing on the ground has the ability to handle firehose 
data streams of low entropy data from the on-orbit sensor. Careful selection of 
downlinked data sets places less stress on expensive downlink and processing facilities. 
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Adaptive processing, then, gives us a two pronged approach to the problem of excessive 
data. First, the technique can be applied onboard to preselect only data of interest for 
further processing, addressing the volume problem at the source. Secondly, adaptive 
computing techniques have been shown to be highly effective in the processing of large 
downlinked data sets, in augmented workstations. 

The Hyperspectral Cloud Cover Assessment Algorithm Design - a candidate for onboard 
implementation 

From Hyperspectral data, the extent of cloud coverage of the Earth's surface can be 
determined. This, in turn, can simplify an onboard decision as to whether to take data 
from instalments that cannot see through the cloud cover. Alternately, the percentage of 
cloud cover may also be a metric of interest, and certain observations may actually 
required cloud cover. The key is an algorithm to distinguish and quantify the visible 
transparency of the atmosphere. 

Multispectral algorithms have been applied to limited (3-10) bands of sensed data since 
the early 1960's, with success severely limited by the processing power that could be 
brought to bear, and the data transmission bandwidths available. Data volume reduction 
was not possible, as there was no effective mechanism for source selection of the raw 
data. Multispectral instruments are usually intended for very specific uses. Hyperspectral 
bandwidth, however, leads to more generalized applications. 

The Onboard Processing Challenge 

It has become clear that a cloud cover assessment utilizing hyperspectral data requires a 
tremendous (number) amount of processing power. The current processing technologies 
(cite) being utilized to design spacecraft computing systems are not capable of 
performing such demanding a task. New approaches must be devised to enable the 
processing of a cloud cover assessment algorithm for selection of relevant data for direct 
broadcast to ground users. One of the solutions would be the utilization of adaptive 
computing technology. Adaptive computing, also known as adaptive computing, is an 
emerging technology that utilizes Field Programmable Gate Arrays (FPGA) to implement 
computation intensive algorithms at the gate level. FPGAs are logic devices that offer in- 
circuit re -programmability. In contrast with the traditional Von-Neumann architectures 
found on microprocessors, which provide a general purpose solution for all algorithms, 
an FPGA based architecture allows a developer to design a machine for each algorithm. 
The elegance of adaptive computing is that by implementing algorithms at the gate level, 
acceleration rates of several orders of magnitude faster than current computers are 
attainable. 

A current limitation to the usage of adaptive computing for onboard processing is the lack 
of radiation hard FPGA devices that support reconfiguration. However, the Goddard 
Space Flight Center has started a program to qualify such devices for space applications. 
A contract was awarded to Honeywell, Inc. on January 30, 1998 to qualify the production 
process of the ATMEL AT6010 adaptive FPGA. Xilinx, the leading manufacturer of 
FPGA devices, has started a program for military and aerospace qualification of its 
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manufacturing process. It is a fair expectation that in the near future, 2 to 3 years, the 
availability of radiation hard FPGAs will enable the utilization of adaptive computing for 
spacecraft onboard processing. 

Adaptive computing is in its early stages of development. The technology has been 
enabled with the creation of adaptive FPGA devices by Xilinx, Inc. in 1985. Since then, 
FPGA device architectures have improved considerably. The application of FPGAs in 
fields such as signal processing, communications, and networking have provided useful 
feedback to the manufacturers which in turn modify the architectures and the design tools 
to improve the performance of their devices and facilitate the application development 
process. A considerable knowledge on the efficient utilization of FPGAs has been created 
by device manufacturers, design tool vendors, and application developers alike. A long 
way is still ahead for the full exploitation of the potential of FPGAs in logic design. 
Adaptive computing itself is considered a small segment of the FPGA market, hardly 
representing an income to the device manufacturers. It is seen, however, as the segment 
with the largest potential for the establishment of FPGAs as mainstream computing 
elements. A number of applications have been developed utilizing adaptive computing, 
showing several orders of magnitude acceleration over microprocessor based solutions. 

FPGA devices are classified according to their gate density and clock operational 
frequency. The gate count indicates the logic capacity of the device, or how large an 
algorithm can be implemented. The clock frequency determines how fast a logic circuit 
can operate, establishing an unit of measurement of the design performance. The FPGA 
technology development yields ever fast and denser devices. As a consequence, larger 
systems with higher performance become feasible. 

An alternative to waiting for the technology development is to combine several FPGA 
devices in a programmable interconnection network, thus increasing the capability to 
implement larger algorithms. Such systems are defined as FPGA computers. There are 
several vendors of FPGA computers, each with a proprietary interconnection scheme 
derived from the vendors experience in the implementation of applications. As it is well 
known from the experience of connecting several microprocessors to compose a parallel 
processing system, there is no optimum implementation that will yield the best 
performance for all algorithms. As a result, certain algorithms will perform best in certain 
architectures. In addition, the fast development of FPGA device technology requires a 
constant upgrade of the computer system architecture, sometimes requiring FPGA 
computer vendors to completely overhaul their systems. The consequence of this rapid 
development is that applications need to be re-implemented in order to take full 
advantage of the new architectures. 

Application development is so far the most complex task of all. It is widely accepted that 
there is no single professional with such a wide set of skills required to implement an 
algorithm in a reasonable amount of time. The solution is the same adopted in building a 
spacecraft: compose a team of engineers that covers all areas of expertise required. 
Currently, the required skills range from algorithm analysis to complex logic design, 
VHDL and FPGA design, the understanding of several FPGA computer architectures, 
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and software engineering. A common misunderstanding among engineers is to think that 
the task fits either on the software or the hardware spectrum of engineering skills. It is 
actually a combination of both, or better yet, adaptive computing requires a computer 
engineer with a solid background on both hardware and software development. The 
current state of the technology requires a hardware engineering approach to the design of 
an application. As the technology develops, however, higher level design tools will allow 
an application development approach closer to the software development style. 

Adaptive computing represents a paradigm shift in computer design. It requires a new 
mind set, free from the fetch-execute computing style dictated by the Von-Neumann 
architectures found on microprocessors. As such, a new set of design tools that are 
neither traditional software and hardware oriented need to be devised. This is probably 
the time to address the application development process parting from the physics of the 
problem with no need to sequentialize an algorithm to fit an already established machine 
paradigm. The effort required to pursue such a new computing paradigm is only justified 
by the fact that the underlying fabrication technology of both FPGA and microprocessor 
devices are the same. Advances in the VLSI circuit fabrication process benefit both sides 
in the same proportion. As a consequence, it is possible to maintain the acceleration rates 
of several orders of magnitude faster than microprocessors attainable today by FPGA 
computers. Some may even argue that the more linear structure of FPGAs against the 
function oriented structure of microprocessors will benefit the most from the 
advancement of the VLSI fabrication process leading to an even larger performance gap 
to the advantage of FPGAs. 

The adaptive solution, where a specific algorithm for data reduction is "compiled" 
directly into hardware, has been demonstrated and is viable. 

The Adaptive Computing Process 

The utilization of adaptive computing for spacecraft on-board processing presents the 
same potentials and limitations found on ground applications, with a few exceptions. 
First, there is no need to support a legacy application development process for space 
applications since there is no established on-board computing system that can currently 
support the processing power required by applications such as hyperspectral data 
classification. Second, the space qualification of FPGA devices greatly reduces the rate of 
development of the technology as new devices are not available as often as in the ground 
segment. Finally, the algorithms for space applications fall within a well defined 
spectrum of physical problems. The design of a spacecraft adaptive computer thus 
requires a good knowledge of the application domain in order to devise an optimum 
architecture, one that will best fit a large set of applications. It is also necessary to select 
the proper devices for space qualification based on the ability of the FPGA architecture to 
properly support adaptive computing. FPGA device manufacturers design devices to 
support specific segments of the market. There is not a single FPGA commercially 
available today that presents all the features known to be essential to support adaptive 
computing. Only a few offer a subset of these features. The selection of FPGAs for space 
qualification based on their ability to support adaptive computing might as well drive 
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device manufacturers to implement adaptive features in their architectures. The 
utilization of adaptive computing for space applications is desired not only in the 
scientific remote sensing arena, but also in the commercial satellite communications 
market. The earlier represents the avenue for establishing the desired features on the 
FPGA architectures, while the later represents a stronger drive for the manufacturers to 
produce better devices. 

The constrained domain of scientific remote sensing applications, the limited rate of 
technological change due to device qualification, the lack of a established onboard 
computer architecture, and the current application development model are the perfect 
opportunity to enable the development of a set of design tools that depart from the 
traditional hardware and software paradigms. It is necessary to create a design entry 
model that allows scientists, which understand the physics of the application, to transfer 
their knowledge to an environment that allows the computer engineer to easily design and 
implement the adaptive computing solution in a cost effective amount of time. The 
technology could then be transferred to allow the expansion of the tools to support a 
wider set of applications. This is a novel approach that should be pursued and supported 
by NASA. 

Other Potential Applications 

The approach to implementing a hyperspectral algorithm in adaptive components for 
spacecraft onboard use is of course not limited to cloud cover assessment. Any algorithm 
can in theory be compiled into hardware. The current state of the art in radiation- 
hardened gate arrays and compiler tools limit the class of algorithms that can currently be 
implemented in this manner. Currently, algorithms are limited to fixed point calculations, 
involving lengthy scaling analyses. Within a short time frame, floating point calculations 
can be accommodated. In addition, both the gate capacity and cleverness of the toolkits 
are increasing, and valuable design knowledge is being acquired by project teams. 

A sampling of other algorithms that might lend themselves to this approach are discussed 
below. 

Within the Earth Sciences community, there are numerous applications that could benefit 
from the utilization of the technology discussed here. These all involve the onboard 
generation and direct downlink of data products of a local interest in a timely fashion to 
end users in the field. In most cases, the timeliness issue precluded the downlink of data 
to a classical ground processing facility for data product generation, followed by a 
dissemination, possible by re-uplink and rebroadcast. The complexity of the onboard 
system interacts with the complexity, and thus cost, of the ground station equipment. 

Numerous applications exist in the field of Earth Resources mapping, particularly where 
asynchronous events directly affect human activities, or require timely response. In many 
cases, the required assessment, data product calculation, and distribution must be 
performed at the data source. This implies the capability of onboard processing of sensed 
data, and direct downlink of the resultant data products, in parallel with the normal data 
downlink. It is anticipated that several algorithms could reside in onboard memory, and 
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that code could be uplinked rapidly to implement new or modified algorithms in response 
to unanticipated events. 

One potential data product for onboard calculation involves the determination of the 
perimeter of a forest fire, a classic edge-detection problem, given a thermal band image. 
Direct downlink to the forest fire command center is essential for freshness of the data. 
The on-site equipment must be small, lightweight, and inexpensive, so that it can be air- 
dropped and abandoned if necessary. This implies a laptop system, augmented with 
processing power as necessary, with an associated lightweight ground terminal. 

The data products of interest to the fire containment crews on the ground include the 
flame front location, direction and rate of spread, smoke plume dimension, and hot spot 
detection. These data are used for personnel and equipment placement, logistics support 
planning, and safety. Observations from U-2 aircraft as well as TIROS series weather 
satellites have been used to gather relevant data on large scale fires. 

Perimeter determination can be accomplished with a Laplacian operator, which is 
homogeneous. This is followed by a scaling, a full rectification, and a thresholding. The 
onboard data storage requirements can be minimized by clever organization of the 
algorithm. Data storage is a premium item that must be minimized for space flight use. 
Not only are the storage devices expensive, but they consume resources such as size, 
weight, and onboard power. 

Similar to the determination of the perimeter of a fire is the determination of the area of 
an oil slick on a body of water. In this case we are interested in the perimeter, but can 
also determine the extent of the "blob", and track the shape, location, and drift. Downlink 
data can be send directly to the site, and used to position containment equipment. The oil 
slick identification problem maps easily to the problem of determining the extend of the 
spread of floodwaters. 

Another related problem is the timely determination of the location of schools of fish near 
the continental shelves, using ocean colorimetry. This process is of interest to commercial 
fishing fleets, and the timeliness of the information is essential, requiring direct downlink 
to fishing fleets. Cost of the fleet equipment is also an issue. 

Similar applications involve specialized operations to image the plumes of active 
volcanoes, or to locate and track the eye of severe storms (hurricanes or typhoons). All of 
these processes are classical image processing applications that can be hosted on a 
adaptive array onboard the spacecraft. 

Wind Vector Derivation 

Using successive images of cloud formations, properly registered by visible land mass, 
the wind vectors may be inferred by cloud motion. This process involves multispectral 
imaging (visible and infrared) from spacecraft such as SMS and GOES. The derived wind 
information is of importance in global weather pattern understanding and prediction, and 
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is critical to severe storm forecasting. Implementing this process onboard will result in 
the ability to generate an downlink a data product that is of timely interest. 

Essentially the same process with hyperspectral sensing of the ocean surface can be used 
to derive surface wind force and direction, and wave height, key parameters of interest to 
Naval operations and commercial ventures such as moored oil drilling rigs and fishing 
fleets. 

The required derivation of the data on the ground has been classically been difficult and 
time-consuming, and has not been available operationally. The algorithms require quasi- 
real time implementation on large workstation computers augmented with array 
processors. However, specific algorithms can be implemented on adaptive arrays. 

The basis of the process is outlined below: 

Given GOES full resolution image sequences (note: these are 1024 x 1024 x 32 bits): 

1. Register the successive images, then register the infrared with respect to the visible. 
Registration involves the translation and orientation (but not scaling) of images such that 
known features match. This is a 2d registration problem, where the scaling is assumed to 
be the same. (However, see step 4). 

2. Apply a digital filter to the visible images. The paper (ref 8.6-3) discusses 4 simple 
cases: the one dimensional derivative, the Sobel Filter, the compass filter, and the edge 
preserving filter. These are applied to 3x3 regions, except for the one dimension 
derivative, which was applied to a 5x5 pixel region. The edge preserving filter was 
chosen for implementation, based on a comparison of output results. 

3. Apply filtering to infrared band data. This takes the form of data averaging over 
limited neighborhoods. 

4. Resample the infrared data to visible resolution. This uses a four- point approximation 
to sin x/x to essentially magnify the infrared image to the visible image resolution. In the 
specific case, the infrared image was magnified by 4 along scan lines, and by 8 across 
scan lines. 

Cloud Template Matching 

This is a classical image processing function usually performed as a 2 dimensional 
convolution over the image field. The correlation process run time depends on the size of 
the image, the size of the template, and the template type. Larger templates require more 
costly multiply operations. Larger templates provide inherent filtering, and also respond 
to gradual changes. Cloud templates typically are between 25 and 29 pixels. 
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Crop Health Assessment 


Researchers, using remote sensing data, are able to tell the health of crop fields, and the 
degree of moisture in the plants. Excessive moisture, or drought conditions, are evident. 
Insect infestations are also derived from remotely sensed data. Spectral libraries are being 
built to support these investigations, and the practice is expected to become as routine as 
the use of meteorological data from spacecraft. 

Early detection of problems is expected to allow quick action to mediate the problem, by 
applying or removing water, applying the correct amount of nutrients or insecticides in 
the proper areas, and so forth. 

A growing library of algorithms for processing multiband data is evident in both the 
NASA and commercial arenas. Software packages such as ENVI (Environment for 
Visualizing Images) address this problem. A large commercial industry has grown up 
around value-added image analysis, most using neural network classifiers. Relevant data 
is obtained from US and foreign remote sensing spacecraft, aircraft, and even unmanned 
airborne vehicles. 

Instantiation of classification algorithms into onboard adaptive computing systems has 
the advantage of examining the sensor data at the source, and making the decision as to 
which data to downlink for further processing. For example, only data of distressed 
vegetation need be considered. 

Pollution Monitoring 

Specific organic compounds (methyls, ethyls, benzenes, and acetones) in the atmosphere 
can be detected by comparing their characteristic "fingerprints" with stored templates. 
Onboard processing of raw sensed data holds out the possibility of selective downlink, 
keyed by the detection of specific compounds. Much of the detector technology in this 
area was developed for the military, to detect booster rocket plumes. The onboard 
classification algorithm, based in a adaptive architecture, could be used to preselect 
datasets for download and further investigation. 

The Rad Hard Problem 

Demonstration of the adaptive approach to compilation of algorithms for hyperspectral 
data set processing has been done, but this is just one step, necessary but not sufficient to 
deploy the technology onboard the spacecraft, and close to the source of data. A radiation 
tolerant and space rated version of the technology must be available for implementation. 

The tolerance of semiconductor devices to radiation must be examined in the light of 
their damage susceptibility. In general, the problems fall into two broad categories, those 
caused by cumulative dose, and those transient events caused by asynchronous very 
energetic particles, such as those experienced during a period of intense solar flare 
activity. 



Cumulative radiation dose causes a charge trapping in the oxide layers, which manifests 
as a parametric change in the devices. Total dose effects may be a function of the dose 
rate, and annealing of the device may occur, especially at elevated temperatures. The 
gross indication of radiation damage is the increased power consumption of the device 

Single event upsets (seu's) are the response of the device to direct high energy isotropic 
flux, such as cosmic rays, or the secondary effects of high energy particles colliding with 
other matter (such as shielding). Large transient currents may result, causing changes in 
logic state (bit flips), unforeseen operation, device latchup, or burnout. The results on the 
operation of the device are somewhat unpredictable. The LET (linear energy transfer) is a 
measure of the incoming particles' delivery of ionizing energy to the device. 

The solution for flight quality adaptive arrays is a series of gate arrays from Xilinx. 
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Conclusions 


The following are the conclusions of this paper: 

1 . Adaptive computing has already been applied to workstation augmentation for limited 
sets of algorithms, (see http://fpga.gsfc.nasa.gov/asdp/frreport.htm l Adaptive computing as an 
augmentation technique for specific algorithms is applicable to a large set of sensed data 
in the Earth resources area, to implement low cost, application-specific processing 
stations. 

Although our goal is to define an architecture with an order of magnitude performance 
increase over existing onboard computing resources, it is quite feasible that several orders 
of magnitude are possible. With scalable processor/communication resources, the 
hardware can be more appropriately matched to the problem domain, while retaining 
flexibility, redundancy, and reprogrammability. 

2. Scalable adaptive processing techniques are also applicable to a large set of onboard 
processing tasks involving hyperspectral data sets now and in the immediate future. This 
has already been demonstrated in augmented workstations on the ground, for 
multispectral data. This approach will provide the capability to generate and rapidly 
distribute appropriate data products that cannot otherwise be done. A rapid flight 
prototype of this process can be accomplished quickly and inexpensively, building on the 
ground experience. There is a defined path to radiation hardened parts of the type used in 
the prototype, allowing for a natural migration of this approach onboard the spacecraft. 
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